Purpose

Migrate the append/data-evolution global full-text search path from the legacy in-repository Tantivy implementation to the shared paimon-ftindex-core engine.

This lets Rust and DataFusion read the current full-text index archives produced by Java Paimon and PyPaimon while preserving the existing full_text_search(table_name, column_name, query_text, limit) SQL API.

This PR intentionally focuses on the non-primary-key/global-index path. The primary-key full-text and hybrid search work is covered separately by #568.

Brief change log

• Replace the legacy Tantivy archive reader and writer modules with an adapter over paimon-ftindex-core.
• Read full-text entries from snapshot index manifests through a bounded-concurrency SeekRead adapter.
• Support plain-text queries, the Paimon JSON query DSL, Roaring include filters, shard-local/global row-ID translation, and global Top-K merging.
• Align fast, full, and detail search modes with Java and PyPaimon behavior, including raw fallback for uncovered row ranges.
• Filter deleted rows through data-evolution deletion vectors and keep raw fallback scoring in a single BM25 corpus.
• Add core and DataFusion coverage for native index reads, row-range filtering, raw fallback, query normalization, authorization, and branch table queries.

API and compatibility

• The full_text_search(table_name, column_name, query_text, limit) SQL table-function signature is unchanged.
• The feature-gated Rust API changes: FullTextSearch and SearchResult move from paimon::tantivy::full_text_search to paimon::full_text, while the legacy TantivyFullTextReader and TantivyFullTextWriter APIs are removed.
• Plain-text queries remain supported. Callers using advanced Tantivy QueryParser syntax must migrate those queries to the Paimon JSON query DSL.
• Ordinary Paimon table data files are unchanged.
• The fulltext feature now reads the current Paimon full-text archive format. Legacy tantivy-fulltext auxiliary indexes are no longer supported and must be rebuilt in the current format.

Testing

• cargo check -p paimon --features fulltext
• cargo test -p paimon --features fulltext full_text -- --nocapture
• cargo check -p paimon-datafusion --features fulltext
• cargo test -p paimon-datafusion --features fulltext fulltext_tests -- --nocapture
• cargo clippy -p paimon --lib --tests --features fulltext -- -D warnings
• cargo clippy -p paimon-datafusion --lib --tests --features fulltext -- -D warnings
• Full GitHub CI matrix: 13/13 jobs passed after rebasing onto main

Scope and coordination

• Builds on the reader foundation merged in feat(ftindex): read Java full-text archives via paimon-ftindex-core #563.
• Uses the published paimon-ftindex-core 0.1.0 crate.
• Complements, but does not implement, the primary-key full-text and hybrid search scope in feat(ftindex): primary-key full-text and hybrid search read path (full scope) #568.